PHIL101 through the Lens of Text Mining

Philosophy is an important study that provides us with valuable skills in life. It produces better critical thinkers, more creative thinkers, and better communicators. Despite one’s knowledge of philosophy, this analysis aims to give all levels of audience a general picture of philosophy from a new perspective, using text mining and data visualizations from the History of Philosophy dataset. The dataset contains over 300,000 sentences from over 50 texts spanning 13 major schools of philosophy. The represented schools are: Plato, Aristotle, Stoicism, Rationalism, Empiricism, German Idealism, Nietzsche, Communism, Capitalism, Phenomenology, Continental, Analytic, and Feminism. Sentences are units of analysis for this project. We will first take a look at the text used in each school of thought, examine the trend in philosophical speech over time, conduct a sentiment analysis, and finally compare the similarity in phisophical text among different schools. Without further ado, let’s dive in!

library(beeswarm)
library(data.table)
library(dichromat)
library(dplyr)
library(factoextra)
library(fmsb)
library(ggplot2)
library(gridExtra)
library(qdap)
library(quanteda)
library(quanteda.textstats)
library(RANN)
library(RColorBrewer)
library(readtext)
library(rvest)
library(scales)
library(sentimentr)
library(stopwords)
library(stringr)
library(syuzhet)
library(textreuse)
library(tibble)
library(tidytext)
library(tidyverse)
library(tm)
library(topicmodels)
library(wesanderson)
library(wordcloud)
source("../lib/plot_stacked.R")
source("../lib/speech_funcs.R")
source("../lib/helper_funcs.R")

# df <- read.csv("../data/philosophy_data.csv")

# Read data using data.table to decrease runtime
df <- data.table::fread("../output/philosophy_data_table.csv")

df <- df %>%
  group_by(school) %>%
  mutate(mode_date = find_mode(original_publication_date)) %>%
  ungroup() %>%
  mutate(school = ifelse(school == "german_idealism", "German Idealism", str_to_title(school)),
         school_ordered = reorder(factor(school), mode_date, mean, order = TRUE))

palette <- get_palette(length(unique(df$school)))

What is the main philosophy in each school of thought?

For those who have never studied philosophy, or even if you only know Plato and Aristotle, you might start wondering what the other major schools of thought are and what the main idea of phisolophy is behind each school. To answer those questions, I have constructed a series of wordclouds and looked at the most used bigrams by each school from the philosophy text in our data.

par(mfrow = c(4, 4))
i <- 0
for(s in levels(df$school_ordered)) {
  docs <- Corpus(VectorSource(df[df$school == s, ]$sentence_str)) %>%
    tm_map(tolower) %>%
    tm_map(removePunctuation) %>%
    tm_map(removeNumbers) %>%
    tm_map(removeWords, stopwords("english")) %>%
    tm_map(stripWhitespace)
  
  tdm <- TermDocumentMatrix(docs)
  tdm_tidy <- tidy(tdm)
  tdm_overall <- summarise(group_by(tdm_tidy, term), sum(count))
  
  i <- i + 1
  plt_title <- s
  wordcloud(tdm_overall$term, tdm_overall$`sum(count)`,
            scale = c(5, 0.5),
            max.words = 200,
            min.freq = 1,
            random.order = FALSE,
            rot.per = 0.3,
            random.color = FALSE,
            colors = palette[i])
  mtext(plt_title, side = 3)
}

Wordclouds are visual representations of words that give greater prominence to words that appear more frequently. Here, I created a wordcloud of the words most used by the philosophers belonging to each school of thought. Looking at the wordclouds, we can see that there are two types of schools: one frequently used content words relevant to their philosophies and the other frequenly used function words only. For example, Capitalism, Communism, and Feminism are schools that used very descriptive words in their philosophy texts and one could infer the main theme employed by each school:

• Capitalism focuses on “price”, “money”, and “value” as it describes an economic and political system in which a country’s trade and industry are controlled by private owners for profit, rather than by the state
• Communism places emphasis on “labour”, “production”, and “workers” as it describes a society in which all property is publicly owned and each person works and is paid according to their abilities and needs
• Feminism is centered around women’s rights as reflected in the use of words such as “women”, “girl”, “wife”, and possiblly racial discrimination through the use of words such as “black” and “white”

On the other hand, Plato, Aristotle, Rationalism, or Empiricism are schools that didn’t seem to use many descriptive words that represent their philosphies as we mostly see “one”, “will”, “can”, “ideas” in their texts. It could be explained by the fact that these are the schools of thoughts that are older and tend to be more abstract. Specifically, Stoicism’s texts were heavily written in old English, which makes it more difficult to understand and infer what the main theme is behind their philosphy.

df %>%
  unnest_tokens(bigram, sentence_str, token = "ngrams", n = 2) %>%
  separate(bigram, c("word1", "word2"), sep = " ") %>%
  filter(!word1 %in% stopwords::stopwords(source = "snowball")) %>%
  filter(!word2 %in% stopwords::stopwords(source = "snowball")) %>%
  filter(!is.na(word1)) %>%
  filter(!is.na(word2)) %>%
  unite(bigram, word1, word2, sep = " ") %>%
  group_by(school_ordered, bigram) %>%
  summarize(n_bigram = n()) %>%
  slice_max(n_bigram, n = 10) %>%
  mutate(bigram = reorder_within(bigram, n_bigram, school_ordered)) %>%
  ggplot(aes(x = bigram, y = n_bigram, fill = school_ordered)) +
    geom_bar(stat = "identity", show.legend = FALSE) +
    facet_wrap(~school_ordered, scales = "free") +
    coord_flip() +
    scale_x_reordered() +
    scale_fill_manual(values = get_palette(length(unique(df$school)))) +
    labs(title = "Top 10 Bigrams by School", x = element_blank(), y = "Count") +
    theme_classic()

Instead of looking at individual words, we will now look at bigram which is a pair of consecutive units such as letters, syllables, or words. The bigram plot now gives us a more detailed and comprehensive description of the topics discussed in each school of thought. For example, we can see German Idealism is about “self consciousness”, “pure reason”, “moral law”, while Phenomenology is about being “objectively present”, “surrounding world”, and “self evidence”. As bigram works really well to describe the text, we can also look at n-gram which is a collection of n successive items in a text document to better understand the context of the text and perform further text analyses.

How has the speech of philosophy evolved over time?

After looking at the text used by philosophers in the major schools of thought, I would like to examine the trend in philosophy speech over time, specifically the sentence length variation and number of distinct words used in the text. I first derived the most original publication date when each school published their text, then ordered the school by their mode publication date. Note that there were no publications happening between the year 170 and the year 1637.

df %>%
  ggplot(aes(x = sentence_length, fill = school_ordered)) +
    geom_histogram(binwidth = 2, size = 0.25, show.legend = FALSE) +
    xlim(0, 500) +
    facet_grid(rows = vars(school_ordered)) +
    scale_fill_manual(values = palette) +
    labs(title = "Sentence Length by School", x = "Word count", y = "Sentence count") +
    theme_classic()

Looking at the sentence length distribution by each school, we can see most schools frequently used sentences with ~100-150 words in their texts. However, Plato, Nietszche and Analytic were more likely to use short sentences as their sentence length distributions are either J-shaped or right-skewed, whereas Capitalism and German Idealism were more likely to use longer sentences with >200 words as their sentence length distributions appear more normal or uniform.

df_avg_length <- df %>%
  group_by(original_publication_date) %>%
  mutate(avg_length_time = round(mean(sentence_length), 0)) %>%
  ungroup() %>%
  group_by(school_ordered) %>%
  mutate(avg_length_school = round(mean(sentence_length), 0)) %>%
  ungroup() %>%
  distinct(original_publication_date, school_ordered, mode_date, avg_length_time, avg_length_school)
ggplot() +
  geom_line(data = df_avg_length, aes(x = original_publication_date, y = avg_length_time), stat = "identity") +
  geom_bar(data = distinct(df_avg_length, school_ordered, mode_date, avg_length_school), aes(x = mode_date, y = avg_length_school, fill = school_ordered), stat = "identity", width = 5) +
  scale_fill_manual(values = palette) +
  labs(title = "Average Sentence Length", x = "Year", y = "Word count", fill = "School") +
  theme_classic() +
  theme(legend.position = "bottom", legend.direction = "horizontal")

Given the original publication date of the text, I would like to see how the average sentence length has changed over time, corresponding to the school of thought popular at that time. Looking at the overall trend, although it tends to fluctuate towards the later years, its center remain around 100-200 words per sentence. The sentences used by the later schools of thoughts are longer on average, with Capitalism, German Idealism, and Continental having the highest average sentence length, which further confirms our primary observations of sentence length by each school.

df_distinct_word <- df %>%
  unnest_tokens(word, sentence_str) %>%
  filter(!(word %in% stopwords::stopwords(source = "snowball"))) %>%
  group_by(original_publication_date) %>%
  mutate(ndistinct_word_time = length(unique(word))) %>%
  ungroup() %>%
  group_by(school_ordered) %>%
  mutate(ndistinct_word_school = length(unique(word))) %>%
  ungroup() %>%
  distinct(original_publication_date, school_ordered, mode_date, ndistinct_word_time, ndistinct_word_school)
ggplot() +
  geom_line(data = df_distinct_word, aes(x = original_publication_date, y = ndistinct_word_time), stat = "identity") +
  geom_bar(data = distinct(df_distinct_word, school_ordered, mode_date, ndistinct_word_school), aes(x = mode_date, y = ndistinct_word_school, fill = school_ordered), stat = "identity", width = 5) +
  scale_fill_manual(values = palette) +
  labs(title = "Number of Distinct Words", x = "Year", y = "Word count", fill = "School") +
  theme_classic() +
  theme(legend.position = "bottom", legend.direction = "horizontal")

Style of speech is not only reflected through the sentence length, but also through the number of distinct words used. In order to analyze the number of distinct words, I firt tokenized the text, then removed the stop words to extract words that are most important in individual text. The overall number of distinct words aggregated by original publication date tends to decrease over time, from ~20000 to ~10000-12000 distinct words. However, when aggregated by school, the later schools of thought actually seemed more likely to use more distinct words in their text, with German Idealism and Continental having the highest number of distinct words of almost ~30000 words. On the other hand, Stoicism had the lowest number of distinct words, which might link to the fact that it mostly used old English language and therefore its vocabulary might not be as diverse as those of other schools.

What is the sentiment in philosophical text of each school?

Moving on, I would like to conduct a sentiment analysis in order to understand the context of the data and establish a correlation between the given texts. For each extracted sentences, I applied sentiment analysis using NRC sentiment lexicon. Here, we will focus on eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, disgust) and two sentiments (positive and negative).

emotions <- data.table::fread("../output/emotions.csv")
df_emotions <- cbind(df, emotions)

par(mfrow = c(4, 4))
i <- 0
for(s in levels(df$school_ordered)) {
  emotions_by_school <- df_emotions %>%
    mutate(school = ifelse(school == "german_idealism", "German Idealism", str_to_title(school))) %>%
    filter(school == s) %>%
    dplyr::select(anger, anticipation, disgust, fear, joy, sadness, surprise, trust) %>%
    bind_rows(summarise_all(., ~if(is.numeric(.)) sum(.) else "Total")) %>%
    slice_tail(n = 1) %>%
    as.data.frame()
  emotions_by_school <- rbind(rep(max(emotions_by_school), 8), rep(min(emotions_by_school), 8), emotions_by_school)
  
  i <- i + 1
  plt_title <- s
  radarchart(emotions_by_school, title = plt_title,
           pcol = palette[i], pfcol = alpha(palette[i], 0.3), plwd = 2,
           cglcol = "grey", cglty = 1, axislabcol = "grey", caxislabels = seq(0,20,5))
}

Looking at the radar chart of emotions by school, we can see that trust is a dominant emotion in the text of philosophy of all schools of thought, and comes after trust are anticipation and joy. Disgust and surprise tend to be the least popular emotions reflected in most philosophical text. It is interesting to note that Stoicism, Capitalism, Nietzsche and Feminism were the most “joyful” schools, while Communism, Continental and Analytic seemed to be on the more pessimistic end of the spectrum. Specifically, Continental even had significantly higher “fear” compared to other schools, which might be explained by its idea of rejecting the view that the natural sciences are the best or most accurate way of understanding all phenomena.

Putting the sentiments in the context of time, we observe that the sentiment in philosophical text is more likely to be positive and peaks around the 18th century. Of all the emotions, trust remains to be the highest emotion over time. We can also see the pattern that philosophical texts in the olden days seemed to be more neutral, while philosophical texts in the later years seemed to be more emotionally charged as the emotions all have maximum average count during the 18th century.

How similar are the schools of thoughts?

What philosophers have in common is that they all identify problems and look for solutions. Ancient philosophers may have had different, sometimes contradicting theories, but they all actively thought about communal or individual issues and looked for answers. As such, I would like to compare and contrast the different schools of thoughts through both the phisophical text itself and the sentiment reflected in the text written by each school.

corp_sentence <- corpus(df, text_field = "sentence_str")
dfmat_sentence <- corp_sentence %>%
  quanteda::tokens(remove_punct = TRUE, remove_symbols = TRUE) %>% 
  dfm() %>%
  dfm_remove(pattern = stopwords("en"))
# dfmat_author <- dfm_group(dfmat_sentence, groups = author)
# tstat_dist <- as.dist(textstat_dist(dfmat_author))
# author_clust <- hclust(tstat_dist)
# plot(author_clust)
dfmat_school <- dfm_group(dfmat_sentence, groups = school_ordered)
tstat_dist <- as.dist(textstat_dist(dfmat_school))
school_clust <- hclust(tstat_dist)
plot(school_clust)

After tokenizing the sentences, I then computed the similarity distance between different schools based on the document-feature matrix which describes how frequently terms occur in the corpus. From the dendogram above, we can detect three major groups of schools:

• Group 1: German Idealism, Aristotle, and Analytic
• Group 2: Plato, Rationalism, Empiricism, Phenomenology, and Continental
• Group 3: Feminism, Stoicism, Nietzsche, Capitalism and Communism

It is interesting how the algorithm returns such groups as I first thought the ancient schools are more similar to one another in terms of styles and ideas in their philosophical text, and similar for the modern schools. Another way to think about it is the origin of the prominent phisophers in each school of thought. In order to better interpret the groups, it is necessary to have some background knowledge in philosophy and understand how philosophers interact with and influence one another over the course of history.

emotions_by_school <- df_emotions %>%
  group_by(school_ordered) %>%
  summarize(
    anger = mean(anger),
    anticipation = mean(anticipation),
    disgust = mean(disgust),
    fear = mean(fear),
    joy = mean(joy),
    sadness = mean(sadness),
    surprise = mean(surprise),
    trust = mean(trust)
  ) %>%
  as.data.frame()
rownames(emotions_by_school) <- as.character((emotions_by_school[, 1]))
kmeans_res <- kmeans(emotions_by_school[,-1], iter.max = 200, 5)
fviz_cluster(kmeans_res, stand = FALSE, repel = TRUE,
             data = emotions_by_school[,-1], xlab = "", xaxt = "n",
             show.clust.cent = FALSE, ggtheme = theme_classic())

To better understand the similarity among different schools, we can also dissect the philosophical texts based on the sentiments. Here, I computed the average count of each emotion by each school, then applied K-Means clustering to find similar groups of schools. We detect five groups of schools that are similar to one another in terms of sentiment:

• Group 1: German Idealism
• Group 2: Plato, Aristotle, Communism, Phenomenology, and Analytic
• Group 3: Feminism
• Group 4: Rationalism, Empiricism, and Capitalism
• Group 5: Stoicism, Nietzsche, and Continental

The groups now seem to be more aligned with my initial hypothesis as schools of the same period of time are grouped together such as Plato and Aristotle, or Rationalism and Empiricism. The reason Stoicism, Nietzsche and Continental are in the same group might be due to their tendency of anticipation and fear, while Feminism might be a standalone group due to its extreme “joy” emotion. Again, one should further study the history of philosophy to better understand these groups and be able to explain why one school is more or less likely to be similar to another school of thought.

What do we learn so far?

• The modern schools of thoughts tend to use more descriptive words, as opposed to the ancient schools which mostly use function words in their texts.
• The average sentence length remains around 100 words, while the number of distinct words decreases over time. The modern schools of thoughts are also more likely to use longer sentences and more distinct words in their texts.
• Trust, anticipation, and joy are the most dominant emotions in philosophical text. Texts written in the 18th century are the most emotionally charged.
• Different schools of thoughts might be more or less similar to one another depending on their period of time, origin, text, or sentiment.
• Study philosophy because it’s interesting!